Overview Scoring Rules

Bias

should be around 0.5 (between -1 and 1 according to Gunnar, but that seems just wrong)

Sharpness

smaller is better.

CRPS

generalisation of Brier score to continuous variables. Smaller is better.

LogS

Advantage: logarithmic Score penalises underestimating uncertainty heavily. I feel this is what we want.

Drawback: In contrast to the CRPS, the computation of LogS requires a predictive density. An estimatorcan be obtained with classical nonparametric kernel density estimation (KDE, e.g. Silverman1986). However, this estimator is valid only under stringent theoretical assumptions and canbe fragile in practice: If the outcome falls into the tails of the simulated forecast distribution,the estimated score may be highly sensitive to the choice of the bandwidth tuning parameter.In an MCMC context, a mixture-of-parameters estimator that utilizes a simulated sampleof parameter draws rather than draws from the posterior predictive distribution is a better

–> especially problematic I think if we work with traces and only small sample sizes?

Question do we now the posterior distribution of our draws?

DSS

Predictions over time horizons

1 day ahead

1 day ahead prediction

3 days ahead

3 day ahead prediction

7 days ahead

7 day ahead prediction

14 days ahead

14 day ahead prediction

21 days ahead

21 day ahead prediction

Scoring by metrics

Bias

median Bias across time

Bias across different time horizons

Bias across different time horizons

CRPS

median CRPS across time

CRPS across different time horizons

logS

median LogS across time

LogS across different time horizons

DSS

median DSS across time

DSS across different time horizons

Sharpness

median Sharpness across time

Sharpness across different time horizons

Scoring by days

1 day ahead

Conclusion

Across the bench, Sparse AR seems the most reasonable (little bias, ok DSS and LogS)

AR1 seems to be very unconfident (and therefore performs well on LogS) AR1 seems to be downward biased.

Bias across models 1 day ahead

CRPS across models 1 day ahead

LogS across models 1 day ahead

DSS across models 1 day ahead

Sharpness across models 1 day ahead

horizon score model bottom lower median mean upper top sd
1 bias AR 1 0.0000000 0.1800000 0.4000000 0.4208571 0.6400000 0.9600000 0.2805722
1 bias Semi-local linear trend 0.0045000 0.2600000 0.5000000 0.4886122 0.7000000 1.0000000 0.2828632
1 bias Sparse AR 0.0000000 0.2600000 0.4400000 0.4429796 0.6000000 0.9400000 0.2539925
1 bias Student local linear trend 0.0000000 0.2800000 0.4600000 0.4940408 0.7400000 1.0000000 0.2887849
1 crps AR 1 0.0097440 0.0293893 0.0526451 0.0794650 0.0890898 0.3660674 0.0898345
1 crps Semi-local linear trend 0.0084990 0.0180820 0.0310165 0.0582366 0.0626547 0.2951930 0.0804311
1 crps Sparse AR 0.0101793 0.0239739 0.0382058 0.0670498 0.0746289 0.3242686 0.0869147
1 crps Student local linear trend 0.0078844 0.0175479 0.0276261 0.0574862 0.0627260 0.3423243 0.0799965
1 dss AR 1 -6.7070804 -4.2726994 -3.2469240 -2.7868620 -2.0327237 4.1355028 3.5341067
1 dss Semi-local linear trend -6.8608629 -5.4140323 -4.4642409 -3.1949170 -3.0414601 6.9085930 7.2741175
1 dss Sparse AR -6.2707872 -4.7646250 -4.0417495 -3.2725326 -2.6254400 3.2154416 4.1854044
1 dss Student local linear trend -6.9603421 -5.6213290 -4.5726281 -3.0854304 -3.0623496 8.5178083 8.6348196
1 logs AR 1 -2.3759070 -1.2771117 -0.8254659 -0.3512798 -0.2124687 3.7791279 3.3550183
1 logs Semi-local linear trend -2.4113148 -1.7454975 -1.2850647 0.1676840 -0.5942576 5.1365309 14.1584179
1 logs Sparse AR -2.2151640 -1.4355747 -1.0007693 -0.4215960 -0.4075420 2.3876052 4.8899705
1 logs Student local linear trend -2.5628558 -1.8588923 -1.3665089 0.6381316 -0.5829726 7.2153839 21.4649779
1 sharpness AR 1 0.0221096 0.0709010 0.1052553 0.1373040 0.1700109 0.4080446 0.1137602
1 sharpness Semi-local linear trend 0.0220793 0.0425032 0.0709581 0.0970184 0.1132235 0.3527295 0.0872100
1 sharpness Sparse AR 0.0273751 0.0597690 0.1020642 0.1307556 0.1481714 0.4327618 0.1075388
1 sharpness Student local linear trend 0.0201258 0.0389854 0.0597141 0.0908121 0.1074593 0.3644611 0.0861209

3 day ahead

Scoring

Conclusion

AR1 seems very bad in terms of bias and everything.

Sparse AR is the best in terms of crps, AR1 the worst. Sparse AR also best with dss

–> take Sparse AR

All models have a tendency to be downwards biased, the local and semilocal ones tend to do a bit better.

Bias across models 3 day ahead

CRPS across models 3 day ahead

LogS across models 3 day ahead

DSS across models 3 day ahead

Sharpness across models 3 day ahead

horizon score model bottom lower median mean upper top sd
3 bias AR 1 0.0000000 0.0800000 0.3200000 0.3870222 0.6400000 1.0000000 0.3241965
3 bias Semi-local linear trend 0.0000000 0.1800000 0.4600000 0.4798667 0.7750000 1.0000000 0.3306509
3 bias Sparse AR 0.0000000 0.1600000 0.4000000 0.4207111 0.6400000 0.9800000 0.3044165
3 bias Student local linear trend 0.0000000 0.2200000 0.4500000 0.4877333 0.8000000 1.0000000 0.3268705
3 crps AR 1 0.0171687 0.0614331 0.1103108 0.1825217 0.2175170 0.8051064 0.2065734
3 crps Semi-local linear trend 0.0201682 0.0477773 0.0924898 0.1629336 0.1807126 0.8202050 0.2113492
3 crps Sparse AR 0.0206726 0.0506307 0.0928446 0.1560582 0.1823767 0.7172435 0.1885933
3 crps Student local linear trend 0.0163728 0.0477265 0.0863423 0.1690528 0.1992893 0.9294194 0.2305773
3 dss AR 1 -5.9276533 -2.9450615 -1.7709012 0.6555864 -0.2748510 25.7626906 12.1364841
3 dss Semi-local linear trend -5.1714026 -3.4990369 -2.2604842 1.0410709 -0.5360296 22.6703036 19.6737737
3 dss Sparse AR -5.1341712 -3.3923785 -2.2980771 -0.6108254 -0.7036617 16.5860866 7.2176870
3 dss Student local linear trend -5.4740366 -3.4486021 -2.4468675 7.1027040 -0.5750851 33.8384259 128.8999208
3 logs AR 1 -2.0104986 -0.6239897 -0.0319122 Inf 0.7560601 26.4391328 Inf
3 logs Semi-local linear trend -1.6811899 -0.7726919 -0.1932857 Inf 0.7073339 21.2044455 Inf
3 logs Sparse AR -1.7106809 -0.7202203 -0.2495037 1.3074993 0.5516765 16.3867174 9.0427728
3 logs Student local linear trend -1.8849301 -0.7960376 -0.2705402 Inf 0.7166684 71.4039034 Inf
3 sharpness AR 1 0.0259101 0.1093206 0.1716067 0.2187139 0.2829760 0.6810136 0.1803758
3 sharpness Semi-local linear trend 0.0358389 0.0953194 0.1487791 0.1922980 0.2379244 0.6989025 0.1579592
3 sharpness Sparse AR 0.0448981 0.1150896 0.1658032 0.2166656 0.2701764 0.6841201 0.1613564
3 sharpness Student local linear trend 0.0255382 0.0886775 0.1449848 0.1864862 0.2316276 0.6289547 0.1665941

7 days ahead

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

horizon score model bottom lower median mean upper top sd
7 bias AR 1 0.0000000 0.0250000 0.2800000 0.3428108 0.5800000 1.0000000 0.3221013
7 bias Semi-local linear trend 0.0000000 0.1200000 0.4100000 0.4577838 0.8150000 1.0000000 0.3509869
7 bias Sparse AR 0.0000000 0.0650000 0.3600000 0.3907568 0.6400000 1.0000000 0.3230276
7 bias Student local linear trend 0.0000000 0.1400000 0.4400000 0.4797297 0.8200000 1.0000000 0.3453366
7 crps AR 1 0.0224146 0.0939478 0.1810374 0.3033660 0.3944736 1.1650918 0.3233669
7 crps Semi-local linear trend 0.0408044 0.1136825 0.2071928 0.3195261 0.4456299 1.1783405 0.3019611
7 crps Sparse AR 0.0411224 0.0869667 0.1639003 0.2659190 0.3420645 0.9273282 0.2865941
7 crps Student local linear trend 0.0366862 0.1256516 0.2213939 0.3495900 0.4473379 1.3712852 0.3557144
7 dss AR 1 -5.1511479 -1.9594018 -0.9830888 9.6924487 2.0741012 140.8036503 50.5344641
7 dss Semi-local linear trend -3.6826454 -1.9456928 -0.8493304 4.4328664 1.3250952 34.6275300 32.3297634
7 dss Sparse AR -3.7060228 -2.4520010 -1.2548468 2.4033523 1.0483922 34.4984516 12.6638976
7 dss Student local linear trend -3.6042707 -1.6218931 -0.7108610 6.2082494 1.2599417 48.4292621 48.2765050
7 logs AR 1 -1.6846635 -0.1356936 0.4478093 Inf 2.0985518 287.2871496 Inf
7 logs Semi-local linear trend -0.9164213 0.0317563 0.5867681 Inf 1.5800334 188.5458002 Inf
7 logs Sparse AR -0.8712581 -0.2524683 0.3474695 4.8517056 1.3516859 43.3777326 24.7326588
7 logs Student local linear trend -1.0542115 0.1465011 0.6724498 Inf 1.6191972 586.2568172 Inf
7 sharpness AR 1 0.0260282 0.1413464 0.2472176 0.3073751 0.3930559 0.9850034 0.2636063
7 sharpness Semi-local linear trend 0.0000000 0.1636543 0.2814537 0.3272691 0.4235071 0.8840668 0.2361977
7 sharpness Sparse AR 0.0601856 0.1622306 0.2385763 0.2984173 0.3420976 0.8644332 0.2239063
7 sharpness Student local linear trend 0.0000000 0.1693309 0.3359365 0.3761931 0.5000669 1.0860822 0.2989137

14 days ahead

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

horizon score model bottom lower median mean upper top sd

21 days ahead

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

Bias across model forecasts

horizon score model bottom lower median mean upper top sd